The large amounts of information generated daily challenge data handling facilities as never before. In the context of today's information generation, data is being generated at rates perhaps thousands or tens of thousands of times greater than was the data-generation rate in the 1990s. Historically, large volumes of data sparked explosive growth in data communications. Responses to growing amounts of data generation centered on improving the movement of data based in increased transmission data rates to enhance throughput in communication channels. For instance, transmission pipelines grew from a few tens of megabits-per-second (Mb/s) transmission rates to several tens of gigabits-per-second (Gb/s) rates during the 1990s.
In the same period, typical storage devices, such as hard disk drives (HDDs), when amassed in sufficient numbers, might accommodate large volumes of data, but the rates at which data may be stored and retrieved have not scaled at the same rate as the volume of data stored on the devices has increased. Data access rates for HDDs are at similar orders of magnitude today as they were in the 1990s.
Fundamental storage subsystems have not integrated technology to enable scaling of effective data storage at the same rate that data generation is growing. Hence the challenge to systems handling large volumes of data is not likely to be alleviated by the combination of contemporary HDD technology with high-speed data transmission channels. In order to handle and manage big data, information processing facilities will be pressured to utilize larger volumes of storage with higher performance rates for capturing and accessing data.